Skip to content

feat(krakatoa): operate_exclusively toggles krakatoa features#1060

Open
mattac21 wants to merge 45 commits intomainfrom
ma/toggle
Open

feat(krakatoa): operate_exclusively toggles krakatoa features#1060
mattac21 wants to merge 45 commits intomainfrom
ma/toggle

Conversation

@mattac21
Copy link
Contributor

@mattac21 mattac21 commented Mar 9, 2026

Description

This PR adds a flag evm.mempool.operate-exclusively that will enable the KrakatoaMempool features. These features have been pulled out of the ExperimentalEVMMempool and into their own struct in krakatoa_mempool.go.

Mempool related system and integration tests now run two copies, they run once on the ExperimentalEVMMempool + CometBFT flood mempool, and once on the KrakatoaMempool+ CometBFT app mempool.

One slight modification was made to the ExperimentalEVMMempool though, when interacting with the legacypool, both mempools use the lifecycle hooks instead of directly modifying the legacypool. The ExperimentalEVMMempool calls the BroadcastTxFn when a tx is promoted via the onTxPromoted hook, instead of passing the BroadcastTxFn into the legacypool itself.

Closes: STACK-2091


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • tackled an existing issue or discussed with a team member
  • left instructions on how to review the changes
  • targeted the main branch

@codecov
Copy link

codecov bot commented Mar 9, 2026

Codecov Report

❌ Patch coverage is 72.51185% with 116 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.72%. Comparing base (43a1f17) to head (48dd311).

Files with missing lines Patch % Lines
mempool/krakatoa_mempool.go 79.92% 41 Missing and 15 partials ⚠️
mempool/mempool.go 63.23% 12 Missing and 13 partials ⚠️
mempool/txpool/legacypool/legacypool.go 68.42% 2 Missing and 10 partials ⚠️
rpc/backend/backend.go 0.00% 6 Missing ⚠️
server/start.go 0.00% 6 Missing ⚠️
server/server_app_options.go 20.00% 2 Missing and 2 partials ⚠️
rpc/backend/call_tx.go 0.00% 2 Missing ⚠️
rpc/backend/sign_tx.go 0.00% 2 Missing ⚠️
mempool/txpool/txpool.go 90.90% 0 Missing and 1 partial ⚠️
server/json_rpc.go 0.00% 1 Missing ⚠️
... and 1 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1060      +/-   ##
==========================================
- Coverage   65.45%   63.72%   -1.73%     
==========================================
  Files         331      331              
  Lines       23262    23704     +442     
==========================================
- Hits        15225    15106     -119     
- Misses       6894     7144     +250     
- Partials     1143     1454     +311     
Files with missing lines Coverage Δ
mempool/internal/queue/queue.go 71.42% <100.00%> (-19.37%) ⬇️
mempool/recheck_pool.go 84.33% <ø> (ø)
mempool/tx_tracker.go 55.38% <ø> (-6.16%) ⬇️
mempool/txpool/legacypool/tx_store.go 63.29% <100.00%> (-36.71%) ⬇️
server/config/config.go 41.61% <100.00%> (-5.29%) ⬇️
server/flags/flags.go 0.00% <ø> (ø)
mempool/txpool/txpool.go 50.70% <90.90%> (-2.24%) ⬇️
server/json_rpc.go 0.00% <0.00%> (ø)
x/vm/keeper/keeper.go 83.09% <0.00%> (-2.84%) ⬇️
rpc/backend/call_tx.go 47.84% <0.00%> (-13.90%) ⬇️
... and 7 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mattac21 mattac21 marked this pull request as ready for review March 11, 2026 03:57
@greptile-apps
Copy link

greptile-apps bot commented Mar 11, 2026

Greptile Summary

This PR introduces operate_exclusively as a runtime-configurable toggle (via evm.mempool.operate-exclusively in app.toml/flags) that determines which mempool implementation is used. When enabled, the new KrakatoaMempool (renamed from ExperimentalEVMMempool) is activated with its ABCI InsertTx/ReapTxs exclusive mode and cosmos tx recheck support. When disabled, the simplified ExperimentalEVMMempool runs in non-exclusive mode alongside CometBFT's clist-mempool, relying on CheckTx. The EVMMempool field on EVMD is widened to sdkmempool.ExtMempool and the VMKeeperI interface is decoupled via the new NotifiedMempool abstraction.

Key changes:

  • KrakatoaMempool extracted into its own file (krakatoa_mempool.go) with async EVM + cosmos insert queues, recheck, reap list, and tx tracker
  • ExperimentalEVMMempool significantly simplified: recheck, insert queues, txTracker, and reapList removed — now a lighter non-exclusive pool
  • EVMMempoolConfig.OperateExclusively, PendingTxProposalTimeout, and InsertQueueSize fields moved out of EVMMempoolConfig and into KrakatoaMempoolConfig
  • NotifiedMempool interface added to break concrete-type coupling between VMKeeperI and the mempool
  • Integration test suite extended with TestKrakatoaMempoolIntegrationTestSuite targeting the exclusive path

Confidence Score: 3/5

  • The PR is mostly safe structurally, but a logic issue with silently-dropped cosmos tx insertion errors in the exclusive path warrants attention before merging to production.
  • The architecture is well-structured and the feature toggle pattern is clean. However, in the KrakatoaMempool.InsertAsync path (used by NewInsertTxHandler), cosmos tx validation errors are silently dropped: the cosmosInsertQueue goroutine runs the ante handler asynchronously and any error is sent on a channel that InsertAsync never reads (it takes the default case and returns nil). This causes CometBFT to receive code=OK for invalid cosmos txs in exclusive mode, potentially leading to gossip of invalid transactions. Additionally, ExperimentalEVMMempool loses EVM tx recheck support entirely in this PR (the evmRechecker parameter is removed), which is a non-obvious behavioral regression for the non-exclusive path.
  • Pay close attention to mempool/krakatoa_mempool.go (InsertAsync cosmos error handling) and mempool/mempool.go (loss of recheck in ExperimentalEVMMempool).

Important Files Changed

Filename Overview
evmd/mempool.go Refactored to conditionally set up either KrakatoaMempool (exclusive) or ExperimentalEVMMempool (non-exclusive) based on the new operate-exclusively flag. CheckTxHandler/InsertTxHandler split is intentional per mode.
mempool/krakatoa_mempool.go New file implementing KrakatoaMempool. Contains copy-paste comment errors on onEVMTxPromoted/onEVMTxRemoved, a dead-code operateExclusively field (always true, never read by IsExclusive), and cosmos tx insertion errors are silently dropped in InsertAsync path.
mempool/mempool.go ExperimentalEVMMempool (non-exclusive mode) now simplified: removed recheck support (no evmRechecker/cosmosRechecker), insert queues, txTracker, and reapList. Contains misleading log field "error" in cosmos tx insertion path. Significant behavioral regression from old code.
evmd/tests/integration/create_app.go Adds exclusiveMempool param to CreateEvmd and NewAppOptionsWithFlagHomeAndChainID. Contains typo "exlcusiveMempool" in NewAppOptionsWithFlagHomeAndChainID parameter name.
server/server_app_options.go Adds GetShouldOperateExclusively helper reading from EVMMempoolOperateExclusively flag. Consistent with existing GetPendingTxProposalTimeout and GetMempoolInsertQueueSize patterns.
mempool/interface.go Introduces NotifiedMempool interface (HasEventBus + GetBlockchain) to decouple VMKeeperI from the concrete ExperimentalEVMMempool type. Clean abstraction for polymorphism.
evmd/app.go EVMMempool field type widened from *ExperimentalEVMMempool to sdkmempool.ExtMempool, enabling both KrakatoaMempool and ExperimentalEVMMempool to be stored.
x/vm/keeper/keeper.go evmMempool field type changed from *ExperimentalEVMMempool to NotifiedMempool interface, consistent with the new interface abstraction.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[configureEVMMempool] --> B{GetShouldOperateExclusively?}

    B -- true --> C[KrakatoaMempool]
    B -- false --> D[ExperimentalEVMMempool]

    C --> E[SetInsertTxHandler]
    C --> F[SetReapTxsHandler]
    C --> G[SetPrepareProposal\nNoCheckTxVerifier]
    C --> H[SetMempool]
    C --> I[NOT SetCheckTxHandler]

    D --> J[SetCheckTxHandler]
    D --> K[SetPrepareProposal\napp verifier]
    D --> L[SetMempool]
    D --> M[NOT SetInsertTxHandler/ReapTxs]

    C --> N[KrakatoaMempool.InsertAsync]
    N --> O{EVM tx?}
    O -- yes --> P[evmInsertQueue async]
    O -- no --> Q[cosmosInsertQueue async\n⚠️ errors silently dropped]

    C --> R[VMKeeperI.SetEvmMempool\nNotifiedMempool interface]
    D --> R
Loading

Comments Outside Diff (1)

  1. evmd/tests/integration/create_app.go, line 237 (link)

    Typo in parameter name

    exlcusiveMempool is misspelled — the l and c are transposed.

Last reviewed commit: a168921

@linear
Copy link

linear bot commented Mar 11, 2026

var ErrQueueFull = errors.New("queue full")

// New creates a new queue.
func New[Tx any](insert func(txs []*Tx) []error, maxSize int, logger log.Logger) *Queue[Tx] {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger was unused

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the name of this mock was wrong (missing the I on the interface)

// The transactions can also be pre-filtered by the dynamic fee components to
// reduce allocations and load on downstream subsystems.
func (pool *LegacyPool) Pending(ctx context.Context, height *big.Int, filter txpool.PendingFilter) map[common.Address][]*txpool.LazyTransaction {
func (pool *LegacyPool) Rechecked(ctx context.Context, height *big.Int, filter txpool.PendingFilter) map[common.Address][]*txpool.LazyTransaction {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ExperimentalEVMMempool calls Pending and the KrakataoMempool calls Rechecked.

// enough for the miner and other APIs to handle large batches of transactions;
// and supports pulling up the entire transaction when really needed.
type LazyTransaction struct {
Pool LazyResolver // Transaction resolver to pull the real transaction up
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused and made the filterAndWrapTxs helper have an extra param, so I removed it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved all krakatoa specific features into here. there is a bit of code duplication between this and the experimental evm mempool still. if we think it is worth breaking the common pieces out into more helper functions or a shared embedded struct, I can address in another pr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of the mempool tests were specific to krakatoa, so moved them to krakatoa_mempool_test.go


// RemoveTx removes a tx from the tx tracker and does not record any metrics as
// it exits the tracker.
func (txt *txTracker) RemoveTx(hash common.Hash) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused

// handler for InsertTx, since the ABCI handler obfuscates the error's
// returned via codes, and we would like to have the full error to
// return to clients.
err := b.Mempool.Insert(b.Application.GetContextForCheckTx(txBytes), cosmosTx)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internally the mempool will use the context on the rechecker to run any validation that is happening during insert tx, so there is no need to specifically get the check context

Copy link
Contributor

@technicallyty technicallyty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although i skimmed a fair chunk here. seems like it was mostly moving stuff around, the feature flag, and then a bunch of stray clean ups. unsure if there is anything delicate that changed. if theres anything you think needs particularly careful attention lmk. otherwise ill approve

mempoolConfig, err := app.createMempoolConfig(appOpts, logger)
if err != nil {
return fmt.Errorf("failed to get mempool config: %w", err)
if server.GetShouldOperateExclusively(appOpts, logger) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we could completely move these out now that we have configs for this and a flag to turn this on and off. Is there any real reason for keeping this setup logic directly in people's applications? This was already here, so maybe better to track in a separate issue if you agree with this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure what you mean by this, where would it move to? fetching the flags and populating the config feels like a responsibility of evmd not the evm library

@vladjdk
Copy link
Member

vladjdk commented Mar 13, 2026

I wanted to add another comment reiterating that I think it's pretty confusing to know when to use "app" vs. "flood" in the mempool, and we should think of a way to enforce one or the other depending on what mempool gets selected

@mattac21
Copy link
Contributor Author

I wanted to add another comment reiterating that I think it's pretty confusing to know when to use "app" vs. "flood" in the mempool, and we should think of a way to enforce one or the other depending on what mempool gets selected

yeah I agree, we'll probably need to make it a change to NewNode in comet in order to enforce a specific mempool, maybe that then checks the config.toml and makes sure what the user configured matches, or it overrides whatever they put.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants